🕷️️ Job Radar • SCRAPING

peopleperhour.com 🟡 2026-05-04

🔹 Robust Scraper System for Monitoring Websites
👤 Client: 🇬🇧 United Kingdom Member since ****
💰 Price: $948
🚩 Problem: Need for a reliable scraper to monitor multiple websites with keyword searches.
📦 Existing: Not specified

Specifications:

[Target] Monitor 100+ websites for specific keywords.
[Method] Support up to 10 concurrent search tabs, perform daily refresh at 00:00 CET, deduplicate results, retain findings for three days unless marked as favorites.
[UI/UX] User-friendly dashboard for managing searches.
[Stack] Python (Scrapy or Beautiful Soup), Selenium for CAPTCHA handling, Redis for caching and deduplication, PostgreSQL for database storage, Celery for task scheduling.
[Security] Handle site security measures, API constraints, and CAPTCHAs.
[Format] JSON for data export, logs in structured format for easy analysis.

Workflow:

1. Set up web scraping framework (Scrapy or Beautiful Soup) with Selenium integration for handling CAPTCHAs.
2. Implement concurrent search tabs using threading or multiprocessing to support up to 10 searches simultaneously.
3. Schedule daily refresh at 00:00 CET using Celery task scheduler.
4. Deduplicate results using Redis cache and store findings in PostgreSQL database for three days unless marked as favorites.
5. Develop a user-friendly dashboard with UI/UX design focusing on ease of use and clear logging of activities.
6. Implement error handling mechanisms to ensure consistent performance and monitoring system to track scrapped sites.